1. Field of the Invention
The present invention relates to a device for detecting the motion vector of an entire picture due to shake or swing of a video camera, in particular, by an image signal, and a device for correcting unnecessary swing of the entire picture due to shake or swing of the video camera.
2. Description of the Prior Art
A conventional image motion vector detecting device is disclosed, for example, in the Japanese Laid-open Patent Sho. 61-269475.
FIG. 1 is a block diagram of an image swing correcting device applying the conventional motion vector detecting device, in which numeral 201 is an image signal input terminal, 202 is a vector detecting circuit in each detecting region, 203 is a correlation value detecting part, 204 is a vector detecting circuit of each region combining the vector detecting circuit 202 and correlation value detecting circuit 203, 205 is a vector reliability judging means, 206 is a motion vector determining means, and 207 is a swing correcting means.
In this and conventional image motion vector detecting device, in the first place, an image signal continuous for at least two fields in time is fed into the input terminal 201. In the vector detecting circuit 202 of each detecting region, as shown in FIG. 2, four vector detecting regions 209, 210, 211, 212 are preliminarily determined in a screen 219, and each detecting region is divided into multiple subregions 215 having a representative point 217 (the number of vector detecting regions may be four or more). Concerning the representative point of each subregion, the difference from the signal of a position 218 deviated by the quantity of a specific range (i,j): imin&lt;i&lt;imax, jmin&lt;j&lt;jmax between two fields is determined as the correlation value, and the sum of the subregions is obtained. The correlation value is calculated as follows. ##EQU1## where .vertline..DELTA.L.vertline. (i,j):
the absolute value of the difference of the signal between the coordinates (i,j) and a representative point
n: number of subregions in one detecting region
Detecting the deviation (i',j') for giving the minimum value of the correlation value, it is taken as a vector 213 of each detecting region. The correlation value detecting part 203 determines the minimum, mean and maximum of the correlation values, and delivers them, together with the motion vector value, to the vector reliability judging means 205 and motion vector determining means 206. A detailed composition of the vector reliability judging means 205 is shown in FIG. 3. In the reliability judging means 205 the minimum, mean and maximum of the correlation values are fed into a comparator 220, and are compared with the predetermined set value Ref(the threshold level of judging condition). In the comparator 220, for example, if the mean is lower than the set value, it is judged that the reliability is lacking (reliability=0), and otherwise the reliability is confirmed (reliability=1). In the motion vector determining means 206, the mean of the vectors in the region judged to be reliable from the minimum, mean and maximum of correlation values of each region is delivered as being determined as the motion vector of the entire screen. FIG. 4 starts with step 221, and step 222 corresponds to step 204 in FIG. 1, step 223 to step 205 in FIG. 1, and step 206 to step 224 in FIG. 1, and a motion vector is delivered at step 225.
The swing correcting means 208 possesses an image signal storing means of one field or more, and by controlling the reading positions of these signals, when an image with a large fluctuation is fed as shown in 226, 227 in FIG. 5, the screen of the image is moved parallel from 229 to 226 in the direction 231 of correcting the fluctuation, on the basis of the motion vector 230 entered from the motion vector determining means 206, with respect to the input image signals of the n-1-th field, the n-th field, and then enlargement of the screen or other operation is manipulated by interpolation or the like, thereby delivering image signals 232, 233 in a size of one screen less in fluctuation.
In the example explained above, the reliability of the vector is judged by the minimum, mean and maximum of the correlation values, but other methods are also known to judge the reliability, for example, by the difference between the minimum of the correlation values and its peripheral point value, that is, the inclination of the periphery of the minimum point, or by the value of the minimum alone, or by the state of the correlation values.
In the hitherto motion vector detecting device composed as described herein, the reliability is judged anyway by the present state of the correlation values such as the minimum, mean and maximum. However, since the correlation value at each time is unstable due to noise or the like, the reliability cannot be correctly judged by the correlation value at each time alone.
Besides, as in the prior art, when judging on the binary basis (0 or 1) to determined the reliability by a predetermined threshold level, if the reliability judgment in a certain range of the threshold level of the judging condition is ambiguous, in other words, if there is an uncertain region of reliability judgement in a width of w1, w2 with respect to the threshold levels of condition 1, condition 2 as shown in FIG. 6, the following case may be considered. Suppose there are two sets of data D1, D2 as shown in FIG. 6. Both D1 and D2 were obtained in the same condition, but were unstable due to noise or the like, and were obtained as different data. By nature, these two are the same data, and are positioned at a very close distance, but in the ambiguous region as for condition 2, and hence D1 is judged to be reliable, and D2 not reliable. As for condition 1, however, the data D1, D2 belong obviously to the reliable region. In this case, therefore, it is correct to judge that both D1 and D2 are reliable, but actually D2 is misjudged to be not reliable.
Due to such cause, the motion vector detecting device of the conventional constitution often misunderstood that a moving object entered the screen, although it was not actually, moving or detected a vector added with the motion of a moving object when the moving object entered the screen. Therefore, the swing correcting device often malfunctioned as the screen was corrected in a wrong direction from the direction intended by the operator as the correction was interrupted or a vector was added when the motion of the moving object was detected.
In the conventional constitution of the motion vector detecting device, moreover, the motion vector detecting device often malfunctioned when a moving object entered the screen because the motion vector of the entire screen was judged from the result of the determination of the reliability in each region. That is, when detection of a moving object is done independently in each region, a sufficient detecting performance of a moving object cannot be obtained.
This point is described in detail below. FIG. 7(a) shows the time course variations of the correlation values when a moving object enters the screen, and FIG. 7 (b), when vibrations of a specific period are applied to a camera. The period indicated by T1 in FIG. 7(a) is the period when the moving object is present, and the period indicated by T2 in FIG. 7(b) is the time of applying vibrations to the camera. To distinguish the motion due to the moving object and the motion due to camera swing, the level of the correlation value is noticed. In the case of the moving object, the correlation value increases largely due to invasion of the moving object, but the fluctuation of the correlation value is small in the case of camera swing. Accordingly, when the correlation value is larger than a certain level, it is regarded to be due to the moving object, and the reliability is judged to be absent. Depending on the circumstances, however, the correlation varies very little due to moving object or the correlation varies very much due to camera swing, and then it is difficult to correctly judge the two in each region.
Incidentally, when a moving object enters the screen, the motion vector added when the motion of the moving object is detected, is described in detail below. In this case, the swing correcting device often malfunctions by correcting the screen in a wrong direction from the direction intended by the operator.
Referring now to the drawings, this is explained more specifically. As shown in FIG. 64, while the video camera is stopped still, suppose a moving object 52 passes through the screen. First, as shown in interval (a) in FIG. 64, when the background and the moving object coexist within a certain detecting region 48, this detecting region 48 is judged to be not reliable by the state of correlation value 53. On the other hand, the still background in a detecting region 49 is desired to be judged reliable by reliability judging means, but depending on the screen appearance of the background, it may be misjudged to be not reliable as the correlation value 54.
Next, as shown in interval (b) in FIG. 64, when occupying the majority of the detecting region 48, the motion vector expressing the motion of the moving object is detected from that detecting region, and a smaller value 55 than the specified value is indicated by the correlation value 53, and the reliability judging means judges reliable. Besides, the background of the detecting region 49 is still judged to be not reliable, and as a result, the motion of the moving object is employed as the motion vector of the entire screen, and when the swing is corrected on the basis of this, the still background is moved, which results in an unnatural image (the operation of the detecting regions 50, 51 is same as that of 48, 49, respectively, and is not explained herein.).
FIG. 8 shows the vector of each detecting region and the motion vector of the entire image at this time. A vector 234 corresponds to the vector of the moving object of 52 of FIG. 64, a vector 235 to the vector of the detecting region of 48 of FIG. 64, a vector 236 to the vector of the detecting region of 49 of FIG. 64, a vector 237 to the motion vector of the entire screen, and a vector 238 to the vector of the background. Intervals (a) to (e) in FIG. 8 correspond to intervals (a) to (e) in FIG. 64. The motion vector detected in intervals (a) to (e) in FIG. 64 is as shown by 237 in FIG. 8, and when the swing is corrected on the basis of this vector, an abnormal correction occurs suddenly in the state of interval (b) in FIG. 8. Furthermore, the vector due to the camera swing may be added to all vectors in FIG. 8, and an abnormal correction may occur due to the same reason.
In the conventional motion vector detecting device, still more, when the screen is swinging and it is being corrected, the reliability judging means suddenly judges not reliable depending on the screen appearance, and 0 vector is often delivered from the vector determining means. As a result, the swing correcting device does not function, and the screen shows a swinging, hard-to-see image.
Practical examples are described below by reference to FIGS. 71 and 9. FIG. 71 shows the screen before correction expressing the camera swing, detecting regions 75, 77, their correlation value 80, detecting regions 76, 78 and their correlation value 81 (the action of the detecting regions 77, 78 is same as that of 75, 76, respectively, and is not explained.). Numeral 79 is an obstacle. To begin with, in intervals (a) and (c) in FIG. 71, depending on the state of the correlation values 80, 81, the reliability is affirmatively judged in the detecting regions 75, 76, and swing may be corrected normally. In interval (b) in FIG. 71, however, depending on the state of the correlation values 80, 81, the detecting regions 75, 76 are suddenly judged to be not reliable, and the motion vector of the entire screen becomes 0 by the vector determining means, and the swing correction is stopped. Finally, as shown in interval (d) in FIG. 71, when both background and moving object are present in a certain detecting region 75, depending on the state of the correlation value 80, the detecting region 75 is judged to be not reliable. On the other hand, the background of the detecting region 76 is judged to be reliable by the reliability judging means, and the motion of the background is employed, and only the screen swing is corrected, and a normal motion is achieved.
FIG. 9 shows the motion vector of the intervals in FIG. 71. In FIG. 9, numeral 241 is a motion vector of the camera, which swings vertically about 0 vector. Numeral 240 is a vector detected from a detecting region 75, and a motion vector 241 due to camera swing, a motion vector of the moving object 79 in FIG. 71 or other wrong vector is detected. Numeral 239 is a vector detected from a detecting region 76, and a motion vector 241 due to camera swing is detected. A vector 242 is a motion vector of the entire screen delivered from a vector determining means 206 in FIG. 1. The intervals (a) to (d) in FIG. 71 correspond to the intervals (a) to (d) in FIG. 81. Same as in FIG. 71, in interval (b), the vectors 239, 240 in the detecting regions become 0, and the motion vector 242 of the entire screen is also 0. As a result, suddenly, the swing is not corrected, and the screen shows a swinging, hard-to-see image.
In the case of multiple vector detecting regions, malfunction of vector detection when a moving objects enters the screen is explained below.
As shown in FIG. 10(a), (b), suppose a moving object (a car) 246 passes through the screen while the video camera is swinging. Numeral 243 is a detecting region, 244 is a motion vector of the background, and 245 is a motion vector of a moving object. The background is judged to be reliable by the reliability judging means, and the correlation value 249 shows a smaller value 250 than the specified value. While the background and moving object coexist, since the correlation value 252 does not possess the specified minimum value, the detecting region is judged to be not reliable (the shaded area in FIG. 10). When the motion of the moving object has occupied the majority of the detecting region, the motion vector 245 expressing the motion of the moving object is detected from that detecting region, and the correlation value 247 shows a smaller value 248 than the specified value, and the reliability is affirmatively judged by the reliability judging means. As a result, both the motion of the moving object and the motion of the background are employed as the motion vectors of the entire screen, and when the swing is corrected accordingly, since the motion of the moving object is contained, an unnatural image is produced.